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Abstract 

In earlier papers we studied direct limits {G,K) = lim (G„, iir„) of two types of Gelfand 
pairs. The first type was that in which the G„/if„ are compact riemannian symmetric spaces. 
The second type was that in which G„ = Nn x Kn with iV„ nilpotent, in other words pairs 
{Gm Kn) for which Gn/Kn is a commutative nilmanifold. In each we worked out a method 
inspired by the Frobenius-Schur Orthogonality Relations to define isometric injections Cm,n : 
L?{Gn/ Kn) ^ L'^{Gm/Km) for 11 and prove that the left regular representation of G on 
the Hilbert space direct limit LF'iG/K) := limL^(G„/if„) is multiplicity-free. This left open 
questions concerning the nature of the elements oi L^{G/K). Here we define spaces A{Gn / Kn ) of 
regular functions on Gn/Kn and injections i^m.n ■ ^(G„/if„) A{Gm/ K^) for to ^ n related 
to restriction by i'm,n(/)|G„/i<'„ = /■ Thus the direct hmit A{G/K) \mi{A{Gnl Kn)TV,n,n\ 
sits as a particular G-submodule of the much larger inverse limit lim{^(G„/-ftr„), restriction}. 
Further, we define a pre Hilbert space structure on A{G/K) derived from that of L'^{G/K). 
This allows an interpretation of L'^{G/K) as the Hilbert space completion of the concretely 
defined function space A{G/K), and also defines a G-invariant inner product on A{G/K) for 
which the left regular representation of G is multiplicity-free. 

1 Introduction 



Gelfand pairs {G,K), and the corresponding commutative homogeneous spaces G/K generalize 
both the concept of riemannian symmetric space and that of a familiar family of riemannian nil- 
manifolds. Let G be a locally compact topological group and K a compact subgroup. Then (G, K) 
is a Gelfand pair, i.e. L^{K\G/K) is commutative under convolution, if and only if the (left regular) 
representation of G on LP' {G/K) is multiplicity free. 

In two earlier notes ( |W4| and |W5] ) we looked at certain cases where G and K are not locally 
compact, in fact are infinite dimensional of the form lini (Gn, K^) where the {Gn,Kn) are finite 
dimensional Gelfand pairs in the usual sense. We showed in those cases that the multiplicity-free 
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condition is satisfied when Lp'{G/K) is interpreted as a certain Hilbert space direct hmit of the 
usual L?'{Gn/ Kn)- That direct hmit made use of renormahzations inherent in the Frobenius-Schur 
othogonahty relations for matrix coefficients of irreducible unitary representations, so the meaning 
of the elements of L'^{G/K) was not immediate. 

Here we define a ring of regular functions A{G/K) whose nature is transparent. It is a sub- 
algebra of lim^(G„/i^„) where A{Gn/ Kn) is the ring of regular functions in the usual sense and 
lim^(G„/i^„) is defined by restriction of functions. This is accompanied by a G-equivariant injec- 
tive map A{G/K) L?'{G/K) with dense image. That leads to the main results of this note, both 
for the direct limits of compact symmetric spaces of [W4] in Section [3] and for the direct limits of 
commutative nilmanifolds of [W5] in Sections [4] and [5l In both cases the results are that 

A{G/K) injects to a dense subspace of L'^{G/K)^ so L?'{G/K) defines a G-invariant 
(1.1) inner product on A{G/K), the regular representation of G on A{G/K) is unitarized, 
and L^iG/K) can be interpreted as the Hilbert space completion of A{G/K). 

2 The Ring of Regular Functions 

In this section we describe the general setup needed for constructing the ring of regular functions on 
direct systems of Lie groups and commutative spaces, and compare them with direct limits. We 
will specialize to parabolic direct systems of compact Lie groups compact riemannian symmetric 
spaces in Section [3l Then we further specialize the results of this section to direct systems of certain 
classes of nilpotent Lie groups and commutative nilmanifolds in Sections S] and [3 

2A Direct Limit Representations. 

We consider direct limit groups G = lirn Gn and direct limit unitary representations vr = lim7r„ of 
them. This means that tt^ is a unitary representation of G„ on a Hilbert space TL-w^-, that the TL-k^ 
form a direct system whose maps are unitary, and that vr is the representation of G on 7i-„: = lim'H^^ 
given by 

(2.1) T^{g)v = 'nn{gn)Vn for n » so that Vn^V and G„ ^ G send Vn ^ v and Qn — > g- 
This formal definition amounts to saying that vr is a well defined unitary representation of G on 

If u,v £ Ti-nn have the matrix coefRcient 

(2.2) fu,v,n : Gn^C defined by fu,v,n{g) = {u, T^n{g)v)H^^ ■ 

Since TL-w^ — > "^^tt™ is a G„-equivariant unitary injection for m ^ n, we may view TLtt^ as a subspace 
of H-rr^ ■ This done, we have 

(2.3) {u,TTn{g)v)n^^ = {u,'Krn{g)v)n^^ for u,v £ and g e Gn, 
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in other words fu,v,n = fu,v,m\G„ for u,v ^ 'Hn„- Now these coefficients form a direct system. 
Formahy, we have 

A{7rn) '■= {finite hnear combinations of the fu,v,n where u,v £ Ti-nn} 
•^i'^n) •^{T^m) using (j2.3p . and ^(vr) := lim^(7r„). 

In words, ^(vr„) is the space of regular functions on G„ defined by 7r„ and A{7r) is the space of 
regular functions on G defined by vr. They correspond to the idea of trigonometric polynomials 
inside of the circle, or more generally to the idea of Harish-Chandra module. 

Note that A{'Jt) is contained in the projective limit lim^(7r„) (defined by restriction of functions), 
but it is much smaller. 



2B Square Integrable Direct Limits 

From now on we assume that the groups G„ are separable, unimodular, locally compact and of 
Type I. (We will be dealing with commutative spaces Gn/Kn, and the commutativity implies 
unimodularity for Gn)- Then one has the classical decomposition 

(2.5) L\Gn)= / (W.„§KJdK) 

J Gn 

where (i(vr„) is Plancherel measure on the unitary dual Gn and the projective tensor product 
'Htt^'Si'H*^ contains w4(7r„) as a dense subspace. This expresses a function / e L^{Gn) n L'^{Gn) in 
the form f(g) = 7rn(/)d(7r„) where 7r„(/) = f^^^^ f{g)-Kn{g)dg £ 'Hn„§>'H'^^ is a Hilbert-Schmidt 
operator on "H^^ and where one has the Plancherel formula II/II2 = Jq- W'^niDW'Hs'^i'^n)- 

Fix a closed subgroup Z„ in G„ that is co-compact in the center. It can be {1} if G„ has 
compact center. If x £ -^n let Gn,x = {t^u G Gn \ 7^n{xz) = x{z)~^7^nix)} and consider the Hilbert 
space L'^{Gn,x) °^ sections of the G„-bundle (G„ x-^ C) Gn/Zn- Then 

(2.6) L\Gn,x) = /_(W-n§KJ<xK) 

where (in,x('^n) Plancherel measure on Gn,x- The decomposition (j2.5p then decomposes further 
into 

(2.7) L\Gn) = i L\Gn,x)dnix) 

J Zn 

An irreducible unitary representation 7r„ G Gn is called square integrable if its coefficients fu,v,n 
satisfy \ fu,v,n\ £ L'^{Gn/ Zn)- This makes sense because Zn is co-compact in the center of Gn- Then 
one has the Godement-Frobenius-Schur orthogonality relations. In particular there is a number 
deg 7r„ > (called the formal degree) such that 
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Now suppose that we have the G„ arranged so that we have direct systems {vr„} of unitary 
representations and, for each n, every 7r„ G G„ belongs to exactly one of those direct systems. 
Write Pm,n,TT ■ "^TTn — > "^TTm for the direct system maps on the representation spaces. For each such 
direct system we have the direct limit representation vr = lim 7r„ of G = lim G„ and the direct limit 
representation space TCt^ = lini |'H7i-„, /^m.n.Trj- 

We now make two strong assumptions on the system {Gn}- First, suppose that we can (and 
do) choose the co-compact closed central subgroups Z„ C G„ so that 

(2.9) Gn ^ Gn+i maps Z„ = Zn+i ■ 

Then we can write Z for each of the groups Zn, and each of the direct systems {vr„} has a common 
central character x G Z. We now make the further assumption on the groups Gn that 

(2.10) Plancherel-almost~all 7r„ G G„ are square integrable: their \fu,v,n\ S L^iGn/Z). 

As described in |W4j and |W5j we use and (|2.1U|) to scale the inclusions H-^^^Hl^ ^ 

Wtt^iSD'H*^ for m ^ by means of formal degrees and obtain {Gn x G„)-equivariant isometric 
inclusions 

(2.11) Cm,n,. ■■ W.„§K„ ^ n^3K^ defined by ^ (^) fu,.,m. 

In other words Cm,n,-K = ^^^{Pm,n,-K ® Pm,n,TT*)- That gives us direct systems of Hilbert spaces 
and the direct limits 

(2.12) = lim{?^^„,/3„,„,^} and H^mi = lim{(W^„gH;j, Cm,n,^}, 

representation spaces for the irreducible unitary representations tt = lim 7r„ of G and vr (8) vr* of 
GxG. ^ 

In order to sum the H-^^^Hl^ and the Htt^HI as in ([2T6l) . to form Lp'iGy-) = limL^(G„,;^), we 
need the direct integral of (j2.6p to be consistent with the rescaling isometrics of ()2.1ip . Specifically 
we need 



(2.13) 



id 



to commute for m ^ n. In the cases studied here, the Gn.x discrete (or even finite), so the 
Lp'{Gn,x) ^'^^ discrete direct sums of irreducible representations. Then there is no consistency 
problem with the rescaling isometries: one simply takes their discrete direct sum. 

Once the 'Hn„'^'H*^ have been summed as in ()2.6p . we need to control the integration over 
the Z in (|2.7p in order to pass to the limit and form Lp'{G) = L'^{Gx)d{x)- Here we need the 
conditions of (|2.9p . that every Zn = Z, and then we need that 

(2.14) for m,n 3> the measures dm and dn on Z are mutually absolutely continuous. 
In the cases studied here condition (j2.14p will be easy to verify. 
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2C Construction of ^(G). 



Our hypothesis (j2.10p of square integrabihty says that Plancherel-almost-all A{7rn) C 'Hj^^^TL^^, 
in fact form a dense subspace there. We want the summation L (Gn.y) ~ Jp ~('^7rn 

(^ni )d 

n,x \'^n ) 

to restrict to a summation of the A{iTn) and form a space A{Gn,x) consisting of regular functions 
on Gn that transform by x- Iii order to do this we must look at the detailed definition of direct 
integral. 



Definition 2.15 Let {Y,M,t) be a measure space. For each y G y let TCy be a separable Hilbert 
space. Fix a family {sa}aeA of maps Y U^ey such that 

(i) Sa{y) G Hy a.e. {Y,M,t), for all a e A, 
(2.16) (ii) y ^ {saiv), Sf3{y))Hy belongs to L^{Y,t), for all a, P G A, and 

(iii) Tiy is the closed span of {sa{y)}aeA a.e. {Y,t). 

Then the (Hilbert space) direct integral defined by the measure space {Y,Ai,T), the family 
{Tly I y £ Y} of Hilbert spaces, and the family {sa}a€A of maps, is the vector space 

7i = Tiy dT{y) : all maps s '.Y ^ I J Tiy such that 
■ yeY 

(i) s{y)(^ny a.e. (y,r), 

(2.17) 

(ii) y ^ {s{y),Saiy))ny is measurable, for each a £ A, and 

(iii) y ^ {s{y),Sa{y))ny belongs to L^{Y,t), for all a G A 



with inner product (s, s') = / {s{y), s' {y))'}^ dT{y). 

Jy 







The inner product of Definition 12 . 1 51 is well defined, and the direct integral W is a Hilbert space. 
Our problem now is to find an appropriate family {sa}a(^A of maps to (J^ ^g- ^(^n) in order to 
put the A{TTn) together to make spaces A{Gn,x) and A{Gn) of regular functions on G„ along the 
lines of a direct integral of Hilbert spaces. In other words we need the conditions that 

(2.18) to form l2(G„,;,) = / {n^^mijd n,xi'^n) we may choose the Sa with SQ,(7r„) G ^(vr„,) 

in order to form the A{Gn,x) ■= f(f~~- •^{'^n)dn,xi'^n), and then we need the conditions 

(2.19) to form L^{Gn) = j^L^{Gn,x)dn{x) we may assume (^^ (""n ® '^*n)dn,x{'^rS) ^ -^(Gn,x)- 

These conditions are automatic when the G„ are compact. In Sections H] and [5] we will verify 
them for the cases where G„ is a connected simply connected nilpotent Lie group that has square 
integrable representations. 
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Under the assumptions ()2.13p and (j2.14p for forming our direct limits of Hilbert spaces, and 
(j2.18p and (j2.19p for restricting to regular functions before the direct limits, we have the spaces 

(2.20) A{Gn) := L[fA{7Tn)dnj7rn))d{x) and A{G) := lim^(G„). 

Here the A{Gn) form a direct system using ordinary restriction of functions, and A{G) is the direct 
limit of that system. Each A{Gn) is a dense subspace of L'^{Gn) but, because of rescaling by the 
Cm,n,n, we do not have .4(G) as a subspace of L'^{G). However there is a very useful relation which 
we now describe. 

2D Comparison of Direct Limits. 

In this section we will assume (j2.13p . (j2.14p . (j2.18p and (j2.19p . Then we have direct systems 
{L^(G„)} — > L'^{G) and {A{Gn)} A{G). Now define (G„ x G„)-equivariant isometric inclusions 

(2.21) r]n ■■ Ai-Kn) Hjr^^nl^ by fu,v,n ^ V'^^S'^n fu,v,n- 



Proposition 2.22 The maps ??„ of (|2.2ip satisfy rim o Cni,n,Tr{fu,v,n) = Cm,n,Tr o 'nn{fu,v,n) and send 
the direct system {A{Gn)} into the direct system {L^(G„)}. That map of direct systems defines a 
{G X G)-equivariant injection 

r, : A{G) ^ L^{G) 

with dense image. In particular rj defines a pre Hilbert space structure on A{G) with completion 
isometric to L^{G). 

Proof. We extract the result from [NRWj Appendix A]. Let C denote the category of topological 
vector spaces and continuous linear maps. We compute r]m° Cm,n,TTifu,v,n) = 'nn{fu,v,n), as asserted, 
and use (|2.13p and (|2.14p to patch the rjn together to form maps rj^ ■ A{Gn) — > L?'{Gn)- The space 
n) carries its usual topology and we give AiGn) the subspace topology. Now view {.A(Gn)} and 
{L^(G„)} as direct systems in the category C. The rj^ define a morphism {A{Gn)} — > {L?'{Gn)} of 
direct systems in C, so the universal property of direct limits gives a morphism rj : lim^{^(Gn)} — >■ 
lim^{L^(Gn)} of the direct limits in C. Note that T}n is injective and (Gn x Gn )-equivariant with 
dense image. It follows that rj is injective and (G x G)-equivariant with dense image. 

The topological vector space direct limit lim^{L^(G„)} has a pre Hilbert space structure given by 
the Hilbert space structures on the L^(Gn), and the Hilbert space direct limit -L^(G) = lim{L^(G„)} 
is its Hilbert space completion. Thus rj : lim^{.A(G„)} lim^{L^(G„)} is in fact a continuous 
linear map, injective and (G x G)-equivariant, from lim^{.A(Gn)} to L^(Gn). Further, our original 
A{G) is obtained from lim^{^(G„)} by forgetting the topology. Thus in particular rj maps A{G) 
into L'^{G), and it is a (G x G)-equivariant injection onto a dense subspace. □ 
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2E Homogeneous Spaces. 

We now consider direct limit homogeneous spaces G/K = limG„/A'„. Specifically, we require that 
G = Vim Gn, that the are compact subgroups of the G„ such that Kn = Gn n Km for m ^ n, 
and that K = limKn- We also assume ([2J3|) . (j2H|) . ([2J8|) and ([2J9]) so that Proposition E^l] 
applies to {Gn} and G. 

Since is compact, L'^{Gn/Kn) = L'^{Gn)^", the subspace consisting of the right-i^^- 
invariant functions on Gn- Of course we can define Lp'{G/K) = Lp'{G)^ , but for comparison 
with A{G/K) we want L'^{G/K) to be of the form lim L^(G„/Er„). Since Kn is compact, (|2.5p 
gives us 

(2.23) L\Gn/Kn) = L^Gnf" = f {n^MKy-) d{^n). 

we view the direct limit maps Pm,n,n '■ "^irn 'Hnm isometric inclusions and arrange the 
L'^iGn/Kn) into a direct system by requiring that 

(2.24) orthogonal projection Pm,n,TT '■ ^tt™ '^tt„ defines a bijective map 

Then every Um S "H^^ has unique expression Um = Un + x with Un G "H^" and x ± Pm,n,Tri'H-n-„)- 
As TTn is irreducible and = H'UnlP + H^^lP we have a constant c = Cm,n,n such that ||un|| = 

c||nm||- Here < c ^ 1 and n„ = Pm,n,TT{um)- Thus we have an isometry 

(2.25) am,n,7r : Ti-^J^ = H^^ by am,n,7r('Un) = Cm,„,,rnm where Un = Pm,n,7T{Um.)- 

Now we have G^-equivariant isometric injections 



(2.26) Cm,n,n : H^MKJ''" ^ ^^^^(K^)'''" defined by ^ c^,„,^ (S^)'^' 

Then | ('H7r„'8'('H*^)^") ,Cm.n,7r| is isomorphic to a subsystem of the Hilbert space direct syst' 
{ {n^„®ni^) , Cm,n,n} of (127121) . As a result we have 

(2.27) n^Kf = hm { {n^MKy") , Cn,n,.} 

in the Hilbert space category, and they fit together under the direct integral (|2.23p to give us 

(2.28) L\G/K) := L^G f = lim {L\Gn/Kn),Cm,n} ■ 

Prom ([231) and ([2:2i]) we have 
(2 29) ~ {finite linear combinations of the fu,vn,n where u £ Ti.Tr„ and Vn S Ti-^J^} and 

t'm.n.TT : Ai-TTn)^" ^ A{TT,n)^"' by fu,Vr^,n ^ fu,Vru,m where projection Pm,n,7T{Vm) = Vn- 

Lemma 2.30 If f e ^(vr^)^" then i^ni,nAf)\Gn/K„ = f- 



em 
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Proof. Let u G H-n^ and w„ G ^^r- Then Vm = Vn + x & with x ± H-,,^. Given 

g £ Gn compute {l^m,n,w{fu,v„,n)) (o) = fu,v„,,m{g) = {u,'Km{g){Vn + x)) = {TTm{g^^)u, Vn + x)) = 
{'^m{g~^)u,Vn) = {u,7rm{g){Vn)) = {u, 7^n{g){Vn)) = fu,Vn,n{9)- ^ 

Equation 12.291 defines a direct system {.4(7r„)^", Mm,n,7r} and Lemma 12.301 shows that its maps 
are inverse to restriction. Thus the direct hmit 

(2.31) ^(tt)^ := lim{^(7r„)^" , Um,n,.] 

sits naturally as a G-submodule of the inverse hmit lim{^(7r„)'^", restriction}. Now, from Propo- 
sition [2?22l we can take right i('-invariants as follows. 

Our assumptions (I2.13P and (I2.14p for forming our direct limits of Hilbert spaces, and (j2.18p 
and (|2.19p for restricting to regular functions before the direct limits, carry (j2.20p over to Kn~ and 
X-invariant regular functions as follows. 



A{Gn/Kn) :=^(G„)^"= / (/ A{T:n)''-dn,^{^n))d{x) 

(2.32) Jz^JGr.,^ ' 



\K 



and A{G/K) := ^(G)^ = lim{^(G„/i^„), z.^,„}. 

In view of Lemma 12.301 the A{Gn/Kn) form a direct system whose maps are inverse to restriction 
of functions and A{G/K) is the direct limit of that system. Each A{Gn/ Kn) is a dense subspace of 
L'^{Gn/ Kn) but, because the Vrn,n are not isometric, we do not have A{G/K) sitting naturally as 
a subspace of L?'{G/K). As in the group level setting we can manage this in a somewhat abstract 
way. The G^-equivariant isometric inclusions restrict to 



(2.33) : ^(7r„)^" ^ W.„§(Kj''" by /„ 

,v,n- 

Proposition 2.34 The maps Cm,n,7r of (|2.26p . i^m,n,n of (|2.29p andrjn^-w of (|2.33p satisfy 

for fu, v,n £ A^TTn)^" ■ Thus they inject the direct system {A{Gn)^" ii^niju} into the direct system 
{L^(G„)'^", Cm,n}- That map of direct systems defines a G -equivariant injection 

?j:A{G/K)^L^{G/K) 

with dense image. In particular rj defines a pre Hilbert space structure on A{G/K) with completion 
isometric to Lp'{G/K). 

Proof. The point is that Cm,n,7r — Cjn,m— l,7r ■ Cm— l,m— 2,7r ' . . . ' Cj^+l,n,7ri SO Cm,n,-K — Cm,l,7r/Cn,l,7r' 

Now compute 



(Cm,n,7r Vn,n^{fu,v„,n) Cm, n, tt (Cn,l,7r \/deg VT^ fu,v„,n') 

1 /2 

— Cm,n,TT ^ deg 7r„ ) Cn,l,7r\/deg TT^ fu,Vm,m 

Our assertions follow as in the proof of Proposition I2.22[ □ 



8 



2F Commutative Spaces. 



Suppose that the Gn/Kn are commutative spaces, i.e. that the {Gn,Kn) are Gelfand pairs. If 
7r„ G Gn then dim'H^" ^ 1, in other words either Ti^^ = or there is a unique (up to scale) 
unit vector w„ G the latter case (j2.24p simply asserts that Vm cannot be orthogonal to 

Pm,n,nivn), SO that Proposition 12.341 is valid. Here there is a problem: given " / and m ^ n 
we must have 'H^J^ 7^ 0. In the remainder of this paper we study two situations in which this 
holds: when the Gn/Kn are compact symmetric spaces, and in many cases where the Gn/Kn are 
commutative nilmanifolds. 



3 Limits of Compact Lie Groups and Compact Symmetric Spaces 

Suppose that G„ is a compact topological group. Then the Peter Weyl Theorem says that L^{Gn) 
is the Hilbert space direct sum of the spaces ^(vr„) of coefficients, 7r„ G G„. In particular one has 
a dense subspace of -L^(G„) given by the algebraic direct sum 

(3.1) A{Gn) = Yl .^A^n). 



When Gn is a compact Lie group, the spectrum of the ring A{Gn) is a linear algebraic group, the 
associated algebraic group of G„ for which A{Gn) is the ring of regular functions. This essentially 
is Tannaka's Theorem; see Chevalley \Ch.\ Chapter 6]. Thus A{Gn) is the ring of regular functions 
on Gn as well as on its associated algebraic group. Note that the associated algebraic group is the 
complexification {Gn)c- 

3A Parabolic Direct Limits of Groups. 

We make the standing assumption for this section and the next that 

the Gn are compact connected Lie groups and 
^ ' {G } is a strict parabohc direct system. 

Here parabolic means that the semisimple part of (Pm,n{Gn) is the semisimple part of the centralizer 
of a toral subgroup of Gm, in other words that ipm,n{[Gm Gn])c is the semisimple component of a 
parabolic subgroup of {Gm)c- Thus we have Cartan subalgebras i)n C 0„ such that dip-m^ni^n) C l)m 
and simple root systems C if}* such that 

(3.3) ii Tp £ then there is a unique ip' £ such that dt/^J^ ^(V"') = V'- 
Now we can enumerate the simple root systems as 

(3.4) = {ipn,i, • • • , tpn/r^} > = rankg„, and ipnj = dip*^^^{ipm,j) for 1 ^ j ^ 4 and m^n. 

Dually we have enumerations of the systems of fundamental weights as 

(3.5) H„ = . . . , in/,,} , 4 = rank0„, and in,j = d^m,n(.^m,j) for 1 ^ j ^ 4 and m^n. 
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Now the correspondence 

hCn,i H h ki„U/„ ^ kiS,m,i H h ke^U,e„ 

sends dominant integral weights to dominant integral weights. Fix an index uq and a dominant 
integral weight A„q for (flno, i)no, ^no)- For n ^ tiq we now have a dominant integral weight for 
(flnj [}n; ^n) and a corresponding direct system of irreducible representations of the Gn given by 

^ ^ {^Tn.A} where A„o = /ci^„o,i H h h Xno/^,, and 

(3.6) ^ " " 

TTn.A e Gn has highest weight A„ = H h kii^^in,i^Q- 

Choose unit highest weight vectors Vn,\ £ Wtt^ . We embed representation spaces 

(3.7) f3m,n,\ = Pm,n,n^ ■ W^r^,;, ^ by X(t;„_A) ^ -'^(t^m,A) for X G l({Qn). 

Then we have the direct limit representation and its representation space: 

(3.8) 7Tx = lim7r„ unitary representation of G = limGn on Tij^^ = lim'H^^ ^ . 
As in (j2.1ip we now have isometric injections 

(3.9) Cn^,n : - W-^,.§K„,. defined by ^ (SS^)'^' 
That gives us direct systems of Hilbert spaces and the direct limits 

(3.10) n^3K, = J, Cm,n}- 

They are the representation spaces for the irreducible unitary representations tta of G and tt\ (8) vr^ 
of G X G. The Peter-Weyl Theorem for Parabolic Direct Limits |W5l Theorem 4.3] exhibits the 
Hilbert space L'^{G) := lim{L^(Gn), Cm,n} as the orthogonal direct sum of the Ht^^^TC^^, and shows 
that the left/right regular representation of G x G on L'^{G) the multiplicity-free discrete direct 
sum of irreducible representations vr^ vr^ of highest weights (A, A*). 

The direct integral conditions (|2.13|) and (|2.14|) are automatic here because the integrals are 
direct sums. Thus, as A varies, the (j3.9p and their limits (j3.10p sum to give us 



1 /2 

(3.11) Cm,n : i'(G„) ^ l2(G„) defined by ^ (SS^ ) ^' ^ ^ 
and the limit Hilbert space 

(3.12) L\G):=\m,{L'{Gn)Xm,n]. 



A 



3B The Ring of Regular Functions for Parabolic Direct Limits. 

As noted earlier, the direct integral conditions (j2.13p . (j2.14p . (j2.18p and ()2.19p are automatic in the 
compact group setting. Thus Proposition 12.221 applies, showing that the A{Gn) and A{G) follow 
the same pattern as in the Peter-Weyl Theorem for Parabolic Direct Limits, and the resulting map 
A{G) L'^{G) defines both a pre Hilbert space structure on A{G) and an interpretation of the 
elements of L'^{G) as the Hilbert space completion of A{G). 
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3C Parabolic Direct Limits of Compact Symmetric Spaces 



Fix a parabolic direct system of compact connected Lie groups G„ and subgroups such that each 
{Gn,Kn) is an irreducible riemannian symmetric pair. Suppose that the corresponding compact 
symmetric spaces M„ = Gn/Kn are simply connected. Up to re-numbering and passage to a 
common cofinal subsequence the only possibilities are 



Compact Irreducible Riemannian Symmetric Spaces Mn — Gn/Kn 


Gn 




RankM„ 


DimM„ 


1 


SU{n) X SU{n) 


diagonal SU{n) 


ri — 1 


- 1 


2 


Spin{2n + 1) x Spin{2n + 1) 


diagonal Spin{2n + 1) 


n 


2n^ + n 


3 


Spin{2n) x Spin{2n) 


diagonal Spin{2n) 


n 


2n^ - n 


4 


Sp{n) X Sp{n) 


diagonal Sp{n) 


n 


2n^ + n 


5 


SU{p + q), p = Pn,q = qn 


S{U{p) X U{q)) 


min(p, q) 


2pq 


6 


SU{n) 


SO{n) 


ri — 1 


(n-l)(n+2) 
2 


7 


SU{2n) 


Sp{n) 


n — 1 


2n^ - n - 1 


8 


SO{p + q), p = Pn,q = qn 


SO{p) X SO{q) 


min(p, q) 


pq 


9 


SO(2n) 


U{n) 




n{n — 1) 


10 


Sp{p + q), p = Pn,q = qn 


Sp{p) X Sp{q) 


min(p, q) 


Apq 


11 


Sp{n) 


U{n) 


n 


n{n + 1) 



These are the cases where the Gn form a parabolic direct system. Now we set about carrying the 
results of Sections I3AI and I3BI from G = lim G„ to M = lim M„ for the systems of Table 13.131 



3D Square Integrable Functions and Regular Functions. 

Recall the decomposition 0n = + under the symmetry 9 of M„. Here Kn is the identity 
component of the fixed point set Gn, in is the +1 eigenspace of d9, and Sn is the —1 eigenspace. 
We assume the alignments 9n = ^m|G„ so t„ C tm and s„ C s^- We recursively choose maximal 
abelian subspaces a„ C 5„ with a„ C and define rrin to be the centralizer of a„ in 6„. For the 
systems of Table [3T3l we have rrin C trim, and we recursively choose Cartan subalgebras in, C rrin 
with that tn C tm- Then the := + are 6'-stable Cartan subalgebras of the Qn- 

The restricted root systems 

A(0n, On) = {q|„„ I a G A(0„, [)„) and Q|a„ 7^ 0} 

have consistent root orders 

if a G A+(s ) and a|a„ / then a\a„ G A+(0 

and similarly we have consistent root orders on the A(m„,t„). Together they define consistent 
positive root systems 

A+(5„,f)n) = {ae A{gn,hn) I either a\a„ € A+(g„,a„), or a|a„ =0 and a|t„ € A+(m„,t„)}. 
Using the parabolic condition we have simple restricted root systems 

^'n = ^'(fln, In) such that if ^ S ^'n then there is a unique ip' G such that ip = V'' I an- 



il 



Now as in (|3.4p we enumerate 

(3.14) = {V'n,!, • • • , i'n/n} , 4 = dim a„, and Vnj = V'mjlan for 1 ^ j ^ 4 and m'^n. 

Using the root orders just described, the Cartan-Helgason Theorem says that the irreducible 
representation ttx of Qn of highest weight A gives a summand of the representation of G„ on L?'{Mn) 
if and only if (i) A|t„ =0, so we may view A as an element of ia* , and (ii) if a G A+(g„, a„) then 
is an integer > 0. 

Condition (i) of the Cartan-Helgason Theorem persists under restriction A ^ A|(j^_-^ because 
tn-i C in- Given (i), condition (ii) says that ^A belongs to the weight lattice of 0„, so its restriction 
to f)n-i exponentiates to a well defined function on the corresponding maximal torus of and 
thus belongs to the weight lattice of 0„_i. Given condition (i), our arrangements C Om and 
tn C tm ensure that condition (ii) persists under restrictions A ^ ^|f)„_i- With this in mind, we 
define linear functionals ^„.j G ia* by 

(3-15) l^^jfej = for l^j^Tn- except that = 2 if 2^^,, G A(0„, a.)- 

The weights ^n.j are the class 1 fundamental highest weights for (0ra,6n) Now as in Section [3AI the 
correspondence 

klin,l H h kl^Cn/^ ^ kiCm,l H h kl>^in/^ 

sends class 1 dominant integral weights to class 1 dominant integral weights and defines direct 
systems of irreducible class 1 representations by 

{^n,A} where A„o = kiino,i H \- h inoU^ and 

(3.16) ^ ° ° 

T^n,\ G Gn has highest weight A„ = /ci^^^i H h ke^^(,n,e„g- 

As before, the corresponding embeddings of representation spaces f3m,n,x ■ "^tt^ x ~^ '^■^n \ ^re given 
by choices of highest weight unit vectors Vn,\ G 'H-,^^ ^ and by X{vn,\) i— > X{vm,\) for X G U{Qn)- 
Again we have the direct limit representations and its representation space 

(3.17) i:\ = limvTn ^j unitary representation of G = limG„ on 7^^^ = lini {'H^^ x-, Pm.,n,\\ ■ 

Note that vr^ is irreducible, has highest weight A, and has a highest weight unit vector v\ = liin Vn.x 
that is invariant under the action of = lirn Kn ■ Given {A„} as in (j3.16p we write TCn.x for Ti-n^ 

Lemma 3.18 For {A„} as in (|3.16p and each n ^ no let Wn,\ he a unit vector in Ti-^y Then 
orthogonal projection Pm,n,x '■ T~(-m,x T~(-n,\ maps w.m,\ to a nonzero multiple Cm,n,\Wn,\ of Wn.x- 



Proof. By G n~equivariance of the projection, Pm,n,\{wm,\) is i^n^nvariant, hence is a multiple of 
Wn^\- Suppose that the multiple is zero, i.e. that Wn^x -L Wm,\- Then w^.x -L "^n.Ai in particular 
Wm,\ is orthogonal to the highest weight A vector Vn^x- As Vm,\ = Vn,x, now Wm,x -L ^m,A- But 
the argument of [Li, Proposition 2.2] shows that Wm,x cannot be orthogonal to Vm,x- We conclude 
Pm,n,x{wm,x) = cWn,x for some nonzero constant c. □ 
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Denote A* = —wo{X) where wq is the longest element of the Weyl group. Then A* is the highest 
weight of the dual vr* ^, and W*^ ^ = Wtt^ ^, ■ Given 7r„^A £ Gn,K we have tt^.a* £ Gn,K and we may 
suppose that its ir„-fixed unit vector wx* is normalized so that the pairing {w\,w\*) = 1. Then 
the space of right-i^„-invariant matrix coefficient functions corresponding to TTn,x is (as a module 
for the left translation action of G„) 

(3.19) = n^^,,(^wx*C = I n G J - 

Lemma 13.181 ensures ()2.24p in our setting, where the regular representations of the G„ on the 
L^(M„) are multiplicity free. Thus we have G„-equivariant injections 

(3.20) am,n,x* : Wj^. ^ Ti^J, , H^-^ = Wn,x*C and Ti^y = Wm,X'C 
as in (j2.25p . and 

1 /2 

(3.21) Cm,n,X ■ 'Hn,\ ® {Wn,X*C) 7im,X O (Wm,A*C) by fu,Wn,n ^ Cm,n,X ( del^^A ) fu,Wm,m 

as in (|2.26p . Now as in Section [2El jWra^A "X* (wn,A*C), Cm,n,A| is isomorphic to a subsystem of the 
system | {T~in,x®T~(-n a) ' Cm,n,A| of (|3.1ip . As a result we have 

(3.22) ?^A®^^^A•C :=lim{(?^„,A®^yn,A*C), lm,n,x] 

and they fit together under the direct integral (j2.23p . which here is reduced to a discrete direct 
sum, to give us L^(M) where M = limM„ as follows. 

(3.23) L'iM) := L^C f = \nR{L\Gn/Kn),Cm,n} . 

We proceed as in Section I2EI but taking advantage of the fact that here the G„ are compact. 
Define 

A{TTn,x)^" = {finite linear combinations of the fu,vn,n where u £ TCtt^ and Vn G W^"^}, 

(3.24) 

,vn,n ' fu,vm,m where projection Pm,n,x{'i^m) — "Wn- 

Thus Lemma [230] says: If / G A{TTn,x)^" then Um,n,x{f)\Gn/K^ = /• 

The ring of regular functions on M„ = Gn/Kn is A{Gn/ Kn) := A{Gn)^" = YliX-^i'^n,x) and 
the i'm,n,x sum to define a direct system {A{Gn/ Kn)-,i'm,n}- Its limit is 

(3.25) A{G/K) := A{Gf = lim{^(G„/i^„), 

As before, the maps of the direct system {A{Gn/ Kn), i^m,n} are inverse to restriction of functions, 
so A{G/K) is a G-submodule of the inverse limit lim{^(G„/^C„); restriction}. 
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Each A[Gn/Kn) is a dense subspace of Lp'iGn/Kn) but, because the i^m,n distort the Hilbert 
space structure, A{G/K) does not sit naturahy as a subspace of L'^{G/K). Thus we use the 
G„-equivariant maps 

where Cm,n,\ = \ \Pm,n,\{wm,\)\\- Now Proposition 12.341 speciahzes to 

Proposition 3.27 The maps Cm,n,\ of (I3.2ip . i'm,n,\ of (13.24P anrf r^^^A of (13.26P satisfy 

m,\ ^ ^m,n,\){fu,v,n) — (Cm,?i,A ^ Vn,x}i,fu,v,n') 

for fu,v,n £ -^(TTn.A)^"- Thus they inject the direct system {A{Gn)^"' ,i^m.,n} irito the direct system 
{L'^{Gn)^" ,Cm,n}- That map of direct systems defines a G-equivariant injection 

?j:A{G/K)^L\G/K) 

with dense image. In particular rj defines a pre Hilbert space structure on A{G/K) with completion 
isometric to L'^{G/K). 

4 Limits of Heisenberg Commutative Spaces. 

We now turn to a class of commutative spaces Gn/Kn on which a nilpotent subgroup of Gn acts 
transitively. Then that nilpotent group must be the nilradical Nn of G^, the isotropy subgroup 
Kn must be a compact group of automorphisms of A^„, and G„ must be the semidirect product 
Nn XI Kn- See |W3t Chapter 13] for an exposition. These spaces usually are weakly symmetric, but 
they are not symmetric. Nevertheless they are accessible because is very similar in structure to 
the Heisenberg group, and the theory is modelled on the Heisenberg group case. Here in Section 
|4]we study the Heisenberg group case, and we examine more general commutative nilmanifolds in 
Section [5l 

4A Regular Functions on Heisenberg Groups. 

We first consider the case where G„ is the Heisenberg group 

Hn = ImC + C" with group composition (z, w){z ,w') = [z + z + Im (w, w'),w + w'). 

There are two sorts of irreducible unitary representations. The ones that annihilate the center 
Z = ImC of Hn are the (1-dimensional) unitary characters on the vector group Hn/Z = M^". The 
ones that are nontrivial on Z, say ■Kn,t with central character C,t{z,w) = e*^, are specified by the 
nonzero real number t and are realized on the Foch space 7in,t of entire holomorphic functions / : 
— > C such that j^^n \f{w)\^e~^^^^'^^'^dw < oo where dw is Lebesgue measure. The representation 
is [TTt{z,v)f]{w) = e*^±*i°i('"-''/2)-^/(i^ - v) where ± is the sign of t/\t\. 

For each real t ^ the representation TTn^t is square integrable (modulo the center of Hn) and 
has formal degree Itl". 
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For each multi-index m = (mi, . . . , rUn), rrii ^ 0, we have the monomial = . . . tw^" . For 
t ^ there is a constant Cn,t > such that the /im := w"^/Vrn\ := {w^^ . . . w^")/ {mil . . . 
satisfy 

f I \ I kii \-i\A Joform/m', 

/ ^m(^^)/^m'(^^)exp(-|t||u;| )dw= \ 
J [1/Cn,t lor m = m. 

Now we normalize the inner product on "Ht^^^^ = Tln,t by 

(4.1) (/,/') = Cn,t J f{w)7^exp{-\t\\w\^)dw. 

Then the monomials form a complete orthonormal set in TCnn t ■ 

In Definition [2TT5] of direct integral now, take (y, A^,r) = (M\{0}, Al, |t|"dt) where A4 consists 
of the Borel sets. For each multiindex m define Sm '■ Y [jT~in,t by Sm(i) = e~l*l;Um- Then 
{sm{t),Sm'{t))Hr.t = e-^l*l5m,m' G L\Y,t). In fact 
(4.2) 

/OO poo poo 

{smit), Sm(i)>w„,, It^dt = 2 e-^'fdt = 2-" / e-2*2"+H"dt = 2~"r(n + 1) = n!/2'^. 
-00 ' Jo Jo 

Thus (|2.16p is satisfied and we have the Hilbert space of (|2.17p : 

/oo 
T-CnAtrdt defined by {R\{0}, M,\t\''dt) and the Sm(t) = e'^^^Hm- 
'OO 

Here \t\'^dt is the Plancherel measure for the Heisenberg group Hn- Since \t\'^dt and dt are mutually 
absolutely continuous on M\{0} we can reformulate ()4.3p as Hn = 'Hn,t dt, which is independent 
of n, for purposes of taking direct limits. 



For multiindices m and m' define rm,m' ■ Y \J'Hn,t®'Hn^t by r^^^t{t) = e~l*l/im ® /w"- 
The /im ® 7w" form a complete orthonormal set in Hn,t'^T~l'^ ^ so (rm,m'(i), '^m",m"'(i))'H„,i = 
e^^'*'(^m,m"'^m',m"', which is inL^(Y',r) as before because J^^{rm,m'it),rm,m'it))H„,t\A'^dt = 2 e~'^^t'^dt 

2^"r(n + 1) = n!/2". Since the 1-dimensional representations in Hn form a set of Plancherel mea- 
sure zero, and \t\'^dt is the restriction of Plancherel measure to {tt^^j | t 7^ 0} this gives us 

/oo 
-00 

where the direct integral is defined by (M \ {0},7W, and the rin,m'(*) = ® Tw"- Now 

we carry this over to rational functions. 



Definition 4.5 For each real polynomial p{t) define Pm,m'(0 = e~l*lp(t)/irn <^ Tw"- -^ei A{Hn) 
denote the set of all finite linear combinations of the Pm,m' with multiplication (p'^ ^/ x p'^ p){t) = 
Ip' {t)p" {t)]{fim+e ® /im'+£')- elements of A{Hn) are the regular functions on Hn and A{Hn) is 
the ring of regular functions on Hn- Note that A{Hn) is isomorphic to the ring of polynomials on 
the complex extension C + C" of the underlying vector space structure of Hn = Im C + C" . 
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Lemma 4.6 The ring A{Hn) of regular functions on Hn is a dense subspace of Lp'{Hn). 

Proof. The pointwise inner product {pm,m'{i),Tm" ,m"i{i))'Hn,t — e~^'*'p(t)(^m,m"^m',m"'- Com- 
puting as before, e'^l*! ji'^l = {k + n)!/2'=+" for integral /fc ^ 0. If p{t) = Y.oPkt'' this 
shows 

/oo 
\e-^\'\p{t)\\t\''dt S J2 + ^)'/2^+" < oo, 

-oo Q 

so each (Pm.m'W, ^m",m"'(*))w„,t ^ LHM\ {0},7W, This proves A{Hn) C L^{Hn). 

Let / G Lp'{Hn) orthogonal to A{Hn)- Denote ft{w) = f{t,w). and r^ ^i/ t = Tm.mK*)- By (|4.4p 
and Fubini, /t is orthogonal to every r^^^' t a.e. t G M. For fixed t, the r^^^i t form a complete 
orthogonal set in TLn,t®'^n,f Thus ft = a.e. t G M. Now / = in L'^{Hn)- We have proved that 
the subspace A{Hn) is dense in L^{Hn)- □ 



4B Functions on the Infinite Heisenberg Group. 

Passage to the limit is easy for Heisenberg groups. The group inclusions are given by (z, w) ^ (z, w), 
identity on the center ImC and the usual C" — > C" by {wi, . . . , Wn) ^ {wi, . . . ,Wn,0, . . . ,0) on 
its complement. Also, and this is a key point, the Hilbert space TCn,t sits naturally in Ti.m,t as the 
closed span of the monomials //m := w"^/Vml for which the exponents m„+i = • • • = = 0. 
That gives us the equivariant isometric inclusions Cm,n ■ Ti-n.t — > 'Hm,t, sending /Xm (as an element 
of TCn,t) to fim (as an element of Ti.m,t)- Now we have 

Hoo = ImC + = \im Hn : infinite Heisenberg group, 

(4.7) Tit = lini Tin.t '■ Hilbert space with complete orthonormal set {fim = w^^/Vuil}, 
TTt = lirn TTn,t '■ irreducible unitary representation of H^o on TCt- 

The space £n,t of matrix coefficients oiTCn,t is spanned by the functions fe^m;n,t '■ 9 ^ {fJ'i, '^n,t{9)lJ'm), 
These coefficients belong to the Hilbert space 

L''{HjlmC;e') := {/ : F„ ^ C | |/| G ^^(i^JImC) and fiz,w) = e-''fiO,w)} 

with inner product {f,f') = j'^n f{z,w)f'{z,w)dw. 

Since iTn,t has formal degree the orthogonality relations say that the inner product in En^t is 
{fi,m;n,t, fe',m';n,t) = |^|~" if ^ = ^' and m = m', otherwise. Now the |t|"''^/f,m;n,t form a complete 
orthonormal set in Sn^t, and £n,t consists of the functions ^n,t,if given by 

(4.8) ^n,tAh) = Yl fi,mmr^^km;nAh) 

where X^^^ \y^£,m{t)\'^ < oo. Thus Lp'{Hn) consists of all functions 

/OO /* OO 

^n,tAh)\tr dt = (V ^e,m{t)\tr/^fi„tih))\trdt 
-oo J-OD ^ ^'"^ ' 
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such that the ipi^m : M ^ C are measurable, m lv'^,m(i)P < oo a.e. t, and J2e in\^i,mit)\'^ S 
Li(M, It^dt). Note that \\^n,^\\l2^H^) = J:e,J\^^,m\\l,^^^\t\„/2aty 

When m ^ n we view an n-tuple m as an m-tuple by appending m — n zeroes; then has 
the same meaning as function on C" and on C". Thus the coefficient function fe^m;n,t '■ Hn C 
is the restriction of ft^Yn;n,t '■ C Now, as in (j2.1ip and (j3.9p . we have isometric {Hn x H^)- 

equivariant injections 

(4.10) Cm,n,t '■ £n,t ^ £m,t by Cm,n,t{\A^^'^ ft,m;n,t) = \t\'^^'^ fi,m;n,t and Cm,ra (^n,(/j) = ^m,|t| ("-™)/2(/p 

of £n,t into £m,t and L^{Hn) into L'^{Hm)- Specifically, Cm,n,t maps a complete orthonormal set in 
£n,t to an orthonormal set in £m,t and Cm,n passes to the direct integral. As expected, the result is 
the multiplicity-free left/right regular representation of H^o x Hoo on 

(4.11) L\Hoo) := lun{L\Hn), Cm,n}. 

Now look back to Definition 14.51 and define 

(4.12) A{Hoo) = lim^(-??n) = {finite linear combinations of the Pm,m'(0 = e~^^^p{t)w'^w'^'}. 

Thus A{Hoo) consists of the finite linear combinations of the e~^^^t'' fim'Pm^, and (for n sufficiently 
large so that /im, fJ-m' S 'Hn,t) 

we have ||6 ^*^i'^/Um /^m' 

'\\l2(^h„) ^ {k + n)l/2^+'^. Consider the maps 

of dMI]): 

7]n,t : A{7rn,t) ^ 'Hn,t®'Hn^t given by rin,t{e~^^^p{t)^JLmV^) = e^^2~i)l*')p(t)^jj^7i^. 
Then Proposition 12.221 tells us that 

Proposition 4.13 The maps r]m,t satisfy r]m,t ° Cm,n,t = Cm,n,t ° iln,t on A{7rn,t) o-nd send the direct 
system {A{Hri)^ into the direct system {L^(i7„)}. That direct system map defines an {Hoo x Hoo)- 
equivariant injection rj : A{Hao) — > L^(i7oo) with dense image. In particular r] defines a pre Hilbert 
space structure on A{Hao) with completion isometric to L?'{Hao)- 

4C Heisenberg Nilmanifolds. 

We first carry the results of Section [2EI and | 4BI over to commutative spaces Gn/Kn where G„ is 
the semidirect product Hn x Kn of a Heisenberg group with a compact group of automorphisms. 
The concrete results in this section will require that Kn be connected and that its action on C" be 
irreducible. 

The classification goes as follows for the cases where Kn is connected and is irreducible on C". 
Carcano's Theorem ([Ca]; or see [BJRl Theorem 4.6] or [Wll Theorem 13.2.2]) says that (G„,K„) 
is a Gelfand pair if and only if the representation of {Kn)ci on polynomials on C", is multiplicity 
free. Those groups were classified by Kac [K, Theorem 3] in another context. His list (as formulated 
in [WHl (13.2.5)]) is 
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(4.14) 



Irreducible connected groups C U{n) multiplicity free on polynomials on 
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77 = 4777 


12 


SU{3) X Sp{m) 


SL{3; C) X Sp{m; C) 


(g) c^™ 


77 = 6m 


13 


U{3) X Sp{m) 


GL{3;C) X Sp{m;C) 


(g) c^™ 


77 = 6777 


14 


C/(4) X 5p(4) 


GL(4; C) X Sp{4; C) 




77 = 32 


15 


SU{m) X 539(4) 


SL{m; C) x Sp{A; C) 




n = 8m, m ^ 3 


16 


U{m) X Sp{4) 


GL{m-, C) X Sp{4; C) 




77 = 8m, m ^ 3 


17 


U{1) X Spin{7) 


C* X 5^171(7; C) 


c« 


77 = 8 


18 


U{1) X Spin{9) 


C* X Spin{9; C) 
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19 


Spin{10) 
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20 


U{1) X Spin{m) 


C* X Spin{10;C) 




n = 16 


21 


C/(l) X G2 


C* X G2,c 




n = 7 


22 


U{1) X Eg 


C* X Ee,c 




n = 27 



Now we have the direct systems 
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1 


SU{n) 




n ^ 2 


2 


U{n) 


C" 


n ^ 1 


3 


Sp{m) 


C" 


n = 2m 


4 


U{1) X Sp{m) 


C" 


n = 2m 


5a 


U{1) X SO{2m) 




n = 2m ^ 2 


5b 


U{1) X SO(2m + l) 


(j~<2m+l 


n = 2m +1^3 


6 


U{m) 


5^(C™) 


m ^ 2, n = ^m{rn + 1) 


7 


SU{m) 


A^(C™) 


m odd, n = \m{m — 1) 


8 


U{m) 


A^(C™) 


n = \m{m — 1) 


9 


SU{e) X SU{m) 




n = (.m, I ^ m 


10 


S{U{£) X U{m)) 




n = im 


11 


U{2) X Sp{m) 




n = 4m 


12 


SU{?>) X Sp{m) 


(8) C^"* 


n = 6m 


13 


?7(3) X Sp{m) 




n = 6m 


15 


SU{m) X Sp{4.) 


C™ c« 


n = 8m, m ^ 3 


16 


U{m) X Sp{A) 




n = 8m, m ^ 3 
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In each case the direct system {Kn} is both strict and parabohc. (We separated entry 5 of Table 
14. 141 into entries 5a and 5b of Table [^TTSl in order to have the parabolic property.) 

We now suppose that {Kn} is one of the strict parabolic direct system given by the rows of 
Table l4l^ 

As U{n) acts on if„ = ImC + C" by k : {z,v) i— > {z, kv) it preserves the equivalence class of 
each of the square integrable representations iTn^t of Hn- The Mackey obstruction vanishes and vr^^j 
extends to a unitary representation tt^ of Hn x U (n) on Ti.n,t- See |WH Section 4] for a geometric 
proof. We will also write tt^ for its restriction, the extension of TTn^t to a unitary representation of 

Gn = Hn XI Kn- 

Let Kn^x denote the irreducible representation of Kn of highest weight A, and its extension 
to Gn = Hn X Kn which annihilates Hn- The corresponding representation space J-n,\ is a finite 
dimensional Hilbert space. Denote Trn,t,x = i^x- Then 7in,t,x '■= Ti-n,t ® ^n,x is its represen- 

tation space. Fix an orthonormal basis {uj} of J-n,x- Then {/im ® Ui\ is a complete orthonormal 
set in TCn,t,x, and we have matrix coefficients 

f£,m,i,j;n,t,x{h, k) = O Ui), ((tT^ ® K^x){h, k)){fi^ ® Uj)) - 

The formal degree deg7r„,t,A = deg(K„,A), so the \t\''^^'^ deg{nn,xy^'^ fe,m,i,j;n,t,x form a complete 
orthonormal set in the space £n,t,x = 'H.n,t,x®'H.nt x matrix coefficient functions. As in (|2.1ip . 
(j3.9p and (j4.10p we have isometric (G„ x G„)-equivariant injections 

(4.16) Cm,n,i,A : fn,t,A ^ £m,t,A by Cm,«,t , A (( 1^ 1" deg A ) ^^^/<!,m,i j ;«,t , A ) = (1*1™ deg K„,A)^^^/£,m,i,i;m,t,A 

Integrate with respect to t and sum on A to construct isometric (G„ x G„)-equivariant injections 

Cm,n '■ L'^iGn) Li'^{Gm)- 

Theorem 4.17 For n > let Kn be a compact connected subgroup of Aut{Hn) such that {Kn} is 
a strict parabolic direct system. Define Gn = Hn x Kn, G = lim{G„} and K = \im{Kn}- Note 
G = Hoo X K. Then {L'^{Gn),Cm,n} is a strict direct system of Hilbert spaces in which the maps 
Cm,n '■ L'^{Gn) —>■ Lp'{Gm) cL^c {Gn X Gn) -cquivariant unitary injections. That gives us the left/right 
regular representation of G x G on the Hilbert space L'^{G) := lim{L^(G„), Cm,n}- Further, that 
left/right regular representation is the multiplicity-free J_^{'^t,x ^"^tx) where iTt^x := lini 7r„^t a- 

Now the construction of A{G) follows the lines of (j4.12p . with properties relative to L'^{G) as 
in Proposition 14.131 As in Definition 14.51 we define 

(4.18) A{G) = lim^(G„) where A{Gn) = {finite linear combinations of the e^'*'p(t)/^^m,jj;n,t,A} 

where p{t) is a real polynomial in t- The argument of Lemma 14.61 shows that A{Gn) is a dense 
subspace of L'^{Gn)- The maps of ()2.2ip in this setting are 

Vn,t,X ■- A{7rn,t,x) ^ Wn,t,A§W*_i A by Vn,t,x{e~\^^Pit) fe,m,i,j;n,t,x) = e*^ 2 "l)!*' y^dcg Kn,X P{t) fi,m,i,j;n,t,X- 

Then Proposition 12.221 tells us that 
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Proposition 4.19 The maps 7]m,t,x satisfy r]m,t,x ° Cm,n,t,\ = Cm,n,t,\ o Vn,t,x on A{TTn,t,x) and 
send the direct system {A{Gn)} into the direct system {L^(G„)}. That system map defines an 
{{H^ X K) X (i^oo ^ K))-equivariant injection rj : A{H^ x K) L'^[H^ x K) with dense image. 
In particular rj defines a pre Hilhert space structure on A(Hoo x K) with completion isometric to 
L^{H^y>K). 

Recall our working hypothesis that {Kn} is one of the 16 systems of Table [^?T5l Since K„) 
is a Gelfand pair with Kn irreducible on C", Carcano's Theorem [Ca] says that the action of Kn on 
the polynomial ring C[C"] is multiplicity free, and it picks out the right i('„-invariants in L^(G„), 
as follows. 

Lemma 4.20 Recall the notation of Section I3AI for A and Kn,\ G Kn- Define G G„ by 
K^x{hk) = i^n,x{k) for h G Hn and k £ Kn- Then 'Kn,t,x '■= TW^t ® has a nonzero Kn-fixed 
vector if and only if i^n x occurs as a subrepresentation of 7Tn^t\K„, and in that case the space of 
Kn-fixed vectors has dimension 1. 

Proof. This is essentially the argument in |Wll Section 14.5A]. Decompose T^t\Kn = S^g7f'^7 7- 
Carcano's Theorem ( [Caj . or see |W31 Theorem 13.2.2]) says that each is either or 1. The 
K,i-fixed vectors oiH® ttvm occur in {rriK* n* ) , and they form a space of dimension m^* . The 
assertion follows. □ 

We view L^(G„/X„) as the space of right-i^T^-invariant functions in With Lemma 

14.201 in mind we set 

K'n = {i^n,x S Kn \ K^j \ occurs in the space of polynomials on C"}. 

Recall that lFn,x denotes the representation space of Given Kn,x £ kJ the right ET^-invariant 
in C[C"] (g) T^ .^ is /^^ ^-(6, ® K*^^^{h*))dk where {hi} is a basis of the K„,A-subspace of C[C"] 
and {6*} is the dual basis of J^nX- Normalize it to a unit vector Wn,t,x- Then the (left regular) 
representation of Gn on L?'{Gn/ Kn) is equivalent to the representation ^ — t Thi^t^K^x dt 

of Gn on f^^{'Hn,t,X ® Wn,t,x'C) dt- 

Proposition 4.21 If m ^ n and Kn.x G Kn then Km,x G Km^ - In that case inclusion C[C"] ^ 
C[C'"] of polynomials maps the highest weight A space for Kn^x onto the the highest weight A space 
for Km,X- 

Proof. The group Kn acts on C" by some representation 7„, so the representation of Kn on 
polynomials of degree d is the symmetric power ^'^(7*). Thus we can compute the set Xn^d of 
highest weights of Kn on the space Pn.d of polynomials of degree d on C". Running through the 16 
cases of Table HTTSl we see that Xn^d C Xm,d- For example (Line 3 of Table ITTSl) the representation 
of Sp{m) on polynomials of degree q in C*^™ is the irreducible representation with highest weight 
lim,!-, and (Lines 5a and 5b of Table HT5]1 the representation of U{1) x SO{n) on polynomials 
of degree q in is the tensor product of the —q^^ power of the usual representation of U{\) by 
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scalars on C" with the multiphcity-free sum of irreducible representations of SO{n) of highest 
weights {Cn,i,3Cn,i,5Cn,i, ■■■,qCn,i} if Q is odd, {0^„,i,2,^„,i,4^„,i,...,g^„,i} if q is even. 

Now let A G Xn^d- Let Vn,\ denote a (nonzero) highest weight A vector for t„ in Pn,d, and 
similarly let Vm,x denote a (nonzero) highest weight A vector for trn- Divide up the variables of C™ to 
{wi, . . . , Wn} for C" and {zn+i-, • • • , Zm} for its complement in C™. Express Vm,x = b bA,BW^z^ 
where each term has total degree |j4| + \B\ = d. Note that Kn treats the Zi as constants. Evaluating 
the Zi at arbitrary constant values C = (c„+i, . . . ,Cm) we have a highest weight A vector for 
By Carcano's Theorem it is a multiple of Vn,x- In other words Vm,x\{z=c} = w-c^n.A- The terms 
&A,_B w"^z^ with z~degree \B\ > yield evaluations of w-degree \A\ < d, and cannot contribute 
to any m^Vn^x- Now fe^^^w^^z^ = whenever \B\ > 0. This shows that Vm,x is a homogeneous 
polynomial of degree d in the wj, as is Vn,x- We conclude that Vjn,x is a nonzero multiple of Vn,x- D 

Corollary 4.22 Condition (|2.24p is satisfied for the direct systems {Kn} of Table 14. 15[ 

Proof. Retain the notation Xn^d for those A such that occurs on the space P^^d of polyno- 
mials of degree d on C". If A ^ Xn^d there are no nonzero i^T^-invariant vectors in Ti.n,t,x, so the 
assertion is vacuous. Now assume A G Xn^d and choose an orthonormal basis {xi, . . . , Xg^n)} of 
the representation space for k„^a in Pn,d- According to Proposition 14.211 that representation space 
is contained in the representation space for Km, A in Pm,d^ so the latter has an orthonormal basis 

{Xi, . . . , Xq(^n)^Xq(^n)+lj • ■ • ) ^^^(m)}- Let {xj, . . . , X*^^yX*^^-^^-^^, . . . , X*^^-^} and {x^, . . . , X*(^n)} 

corresponding dual bases of J^m,x and J^n,x- The if„-invariant vectors in 7in,t,x are the multiples 
of Ylf''^^ Xi® X*, and the i^m-invariant vectors in TLm,t,x are the multiples of Xj x*. The 

adjoint of unitary inclusion is orthogonal projection, which sends Xi x* to Yl'f^^ ^ x*. 

□ 

Let Wn,x be a right-ir„-fixed unit vector in the highest weight A subspace of L?'{Hn x Kn), for 

each Kn^x G -f^n^- Note that Wn,x does not depend on t. Proposition 14.211 says that the inclusion 
^t,x maps Wn,x to a nonzero multiple of Wm,x- Given A we recursively choose the Wn,x 

so that 

(4.23) Wm,X = Cm,n,t,XWn,X + X with X ± Sn,t,X with < Cn,t,X S 1- 

Note Ti-^l, = 'Wn,x*C and 'H^\, = Wm^x*C. Now we have {Hn x -ftTn )-equivariant injections 

(4.24) am,n,t,X* ■ T~(-n,t,X* ~^ ^m^,A*' "m.n.t.A* (^in^A* ) = Cm,n,t,XWm,X* , 

as in (I2.25p . and 

(4.25) Cm,n,t,A : W„,i,A 1^ iwn,t,x*C) Tim,t,x {wm,t,x*C) defined hy f ^ Cm,n,t,xCm,n,t,x{f) 

as in (j2.29p and (j2.3ip . Now as in Section [2El jWn.t.A ® (ty„^A*C), Cm,n,t,x^ is isomorphic to a 
subsystem of the system | {T~Ln,t,x®'H'^ ^ , Cm,,n,t,x^ of (jS.lip . As a result we have 

(4.26) Ht^x ® wx*C := lim ^Tin,t,X O Wn,X*C, Cm.,n,t,X^ 
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and they fit together under the direct integral (j2.23p to give us L?'{{Hn xi Kn)/Kn) as follows. 
(4.27) L^{{H^ X K)/K) := L\H^ x K f = lim [l^{{H„ x K„)/i^„), Cm,n} • 

Combining Theorem 14.171 Lemma 14.201 and Corollary 14.221 we have 

Theorem 4.28 Let A'„)} he one of the direct systems of Table 14.151 Define Gn = Hn x Kn, 
G = lini Gn and K = lini ■ Then (j4.27p is a unitary direct system whose limit Hilhert space is 
G-isometric to L^{G)^ , and the natural unitary representation of G on L'^{G/K) = L?'{G)^ is 
multiplicity free. 

Now we turn to regular functions. As in (j4.12p we define 

A{Hn X Kn) ■= {finite linear combinations of the e~'*'/^^m,jj;n,t,A in £n,t,\}, 

(4.29) A{{Hn X Kn)/Kn) := A{Hn X Knf" = A{Hn X Kn) H S^^^, 

Vm,n,t,\ ■■ A{TTn,t,\)^" ^ A{7:m^t,x)^"' by fu,v,„n ^ fu,v^,m where Pni,n,t,\{Vm) = Vn 

Now we have direct systems and their limits 

A{Hoo K) = lim{^(i?„ X Kn),Cm,n} where Cm,n,t,A : £n,t,x ^ £m,t,\ (|4.16p . and 
(4-30) ^ J. ^ 

A{{H^ X K)/K) = \}x^{A{{Hn X K„)/K„), Vm,n] where Vn,,n,t,X ■■ £n,t,X ^ CT,A ^^M- 

As before, each A{{Hn x Kn)/Kn) is a dense subspace of L?'{{Hn x Kn)/Kn). In order to pass 
this comparison to the limit we use the maps 

(4.31) ^n,t,X ■■ A{7Tn,t,x)^- ^ {Wn,t,X'C) by / ^ C„,i,A 1*1"/' ^deg At„,A /• 

Proposition 4.32 The rjn,t,x satisfy {rjm,t,x ° ^m,n,t,x){f) = {Cm,n,x ° rjn,t,x){f) for f G A{TTn,t,x)^'' 
Thus they inject the direct system {A{[Hn x ii'„)/K„), 1/^,^} of regular functions into the direct 
system {L'^{(Hn>^Kn)/Kn),Cm,n} of square integrable functions. That map of direct systems defines 
an (i/oo XI K)-equivariant injection 

^ : A{{H^ X K)/K) ^ L\{H^ x K)/K) 

with dense image. In particular r/ defines a pre Hilhert space structure on A{{H^ x K)/K) with 
completion isometric to L'^{{Hco x K)/K). 



5 Extension to Commutative Nilmanifolds. 

The results of Section [5] depend on four basic facts. First, the 7r„^t are determined by their central 
character. Second, we have good models Ti.n,t for the representation spaces, such that n does not 
appear explicitly in the formulae for the actions of the group elements. Third, the injections Hn ^ 
Hjn restrict to isomorphisms Z„ = of the centers. And fourth, we have complete information 
on the Plancherel measure for the Hn- In this section we consider a somewhat larger class of 
nilpotent direct systems that satisfy these conditions. In this section we extend our Heisenberg 
group considerations to nilpotent Lie groups with square integrable representations, following the 
general lines of |W3] and |W5] . 
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5A Square Integrable Nilpotent Groups. 

Here is a quick summary of harmonic analysis for connected simply connected groups that admit 
square integrable representations. See |MWj for details, |W3l Section 14.2] for an exposition. Let A'^ 
be a connected, simply connected nilpotent Lie group and n its Lie algebra. Decompose n = 3 + 
and N = Zexp(t)) where 3 is the center of n. Then Z = exp(3) is the center of A^. We say 
that an irreducible unitary representation vr of is square integrable if its coefficient functions 
fu,v{g) = {u,'iT{g)v) satisfy \ fu,v\ G L?'{N/Z). In that case vr is determined by its central character, 
vr = TTj where t G 3* and the central character is exp(^) 1— > e***^^^. In terms of geometric quantization, 
TTf corresponds to the coadjoint orbit in n* consisting of all linear functionals on n whose restriction 
to 3 is t. Further, the antisymmetric bilinear form ht{i^ rj) = rj]) on D is nondegenerate, and (up 
to a positive constant that depends only on normalizations of Haar measures) the formal degree of 
vTj is |Pf(6j)|, where Pf(6t) is the Pfaffiar0 of 6^ : x ^ M. In fact, if tTs is the representation of 
that corresponds to Ad*(A^)s C n*, then vr^ is square integrable if and only if Pf(6s) 7^ 0. In any 
case, 

(5.1) Pf(6t) is a polynomial function of t, Pf(5j) = P{t), and P(0) = 0. 

Again, up to a constant that depends on normalizations, |Pf(6t)| is the Plancherel density. It 
follows that if one irreducible unitary representation of A^ is square integrable then Plancherel- 
almost-all are. In the case of the Heisenberg group Hn, where we identified 3* with R, the Pfaffian 
corresponding to iTn^t is i". 

When we are dealing with a sequence {Nn} if square integrable nilpotent groups, we have to 
keep track of the polynomials (15. ip . so we will write 

(5.2) P{n, t) = Fi{bn,t) '■ corresponding to the group A^^ • 

The point of this, from the viewpoint of commutative spaces, is that many Gelfand pairs are 
of the form (A^ xi K, K) where A^ is a connected simply connected Lie group, K is a. compact 
subgroup of Aut(A^), and A^ has square integrable representations. See |W31 Theorem 14.4.3]. This 
is simplified by the 2-step Nilpotent Theorem [W3t Theorem 13.1.1] of Benson- Jenkins- Ratcliff and 
Vinberg, which says that A^ must be abelian or 2-step nilpotent. In a certain sense representations 
treat those groups as Heisenberg groups: 

Lemma 5.3 ( |W3t Lemma 14.4.1]) Let N he a connected simply connected 2~step nilpotent Lie 
group with 1-dimensional center. Then N is isomorphic to the Heisenberg group Hn where n = 
^(dimj, n — 1), and in particular N has square integrable representations. 

Proposition 5.4 ( |W31 Proposition 14.4.2]) Let N be a connected simply connected 2-step nilpo- 
tent Lie group. Let / G n* such that f\^ 7^ 0. Denote toj = {z G 3 | f{z) = 0} and Wf := exp(tt)j). 
Then 

1. Wf is a closed subgroup of Z, hence a closed normal subgroup of N. 

^Strictly speaking, Pf (bt) depends on a choice of basis of D, for a basis change of determinant at multiphes det bt\r,xa 
by det St and multiphes Pf(&t) by det at. 
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2. The functional f is the puUback of a linear functional f € (n/ttJ/)* and is nonzero on the 
central subalgebra 3/tt)/ o/n/ro/. 

3. The representation [ttj] is the pullback to N of the class [ttji] € N/Wj. 

4. If the representation [vrj] is square integrable then [vrj/] is square integrable, and in that case 
N/Wf has center Z/Wj and is isomorphic to a Heisenberg group Hn where n = ^dim(n/3). 

We now consider a strict direct system {Nn} of 2-step nilpotent connected, simply connected 
Lie groups that have square integrable representations, where the inclusions n„ — > map the 
center 3^ ^ and the complement D„ in decompositions n„ = ln + ^n- Then the direct 

limit algebra n := limn„ has center 3 := lirn g^ and n = 3 + where = lim d„. On the group level, 
Z = lirn Zn = exp(3) is the center of N := lim and we have N = Zexp(d). 

We further assume that the dimensions dim Zn of the centers are bounded. Since they are 
non-decreasing we may assume that they are eventually constant. Passing to a cofinal sequence, 

(5.5) n„ ^ rim maps 3^ = 3m- 

Under that identification we write 3 for all the 3„, 3* for all the 3* , and Z for all the Zn- 
Let t £ 3*. Write bn,t for the bilinear form (^,77) ^ t{[(,i'n]) 

on • Then t corresponds to a 

square integrable representation TTn,t of Nn just when the Pfaffian Pf(6„.t) 7^ 0. For purposes of 
comparing the Pfaffians as n varies, we note that Pf(6„^() is specified by t and a basis of D„, so 
we simply assume that these bases are nested in the sense that the basis of Vn+i consists the basis 
of \3n together with some elements that are 6„+i^j-orthogonal to D„. Thus, if Pf(6„+i^t) 7^ then 
Pf (^n,t) 7^ 0. The converse fails in general, but the following lemma deals with the possibility that 
Ff{bn,t) / = Pf(6m,t). It depends on the fact [MW] that each Ff{bn,t) is a polynomial function 
on 3*. 

Lemma 5.6 Let G 3* denote the zero set of Vi{bn,t) o-nd set a = IJ'^n- Then a is a set of 
Lebesgue measure zero in 3*. 

Proof. Since Nn has square integrable representations, the Pfaffian Pf(6„^f) is a nontrivial poly- 
nomial function of t G 3*, so a„ is a finite union of lower-dimensional subvarieties of 3* . Now the 
set a is a countable union of sets a„ of Lebesgue measure zero. □ 

Define T = {t S 3* | each Pf(6„^() 7^ 0}. So 3* \ a. For every t G T and every index n we have a 
square integrable representation 7r„^j G A^^^. Let t G T, rot = {z G 3 | t{z) = 0} and Wt = exp(tt3t). 
Then Wt is closed in Z, Nn/Wt is isomorphic to a Heisenberg group H^(^n)-, and ■Kn,t factors through 
to the square integrable representation of Nn/Wt with central character e**. As t varies in T the 
TTn,* act on the same Fock space T~id(n),t) d{n) = | dimti„, by formulae independent of d{n). 

We normalize the inner products on the 'Hrf(n),t as before, so the fj.^ form a complete orthonormal 
set, and realize the space £n,t = '^d{n),t®'^*d[n) t niatrix coefficients as the closed span of the 
functions fi^m]n,t '■ g I—*- {fJ'£,'^n,t{g)tJ'm), as in Section m The orthogonality relations say that the 
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inner product on £n,t is given by {f£,m;n,t, fe,m';n,t) = |Pf(^'n,t)| Mf £ = and m = m', and is 
otherwise. Now the |Pf (6n,t)|^/^/<!, m;n,t form a complete orthonormal set in £n t: ciiid as before Sn * 
consists of the functions ^n,t,ip on -ffd(n) given by 

(5.7) ^n,tAh) = 'Pe,m{t)\Piibn,tP^fe,m;n,t{h) with ^ \'Pe,n,{t)f < DO. 

Now L'^{Nn) = £n,t\Pf{bn,t)\dt = Jt ^n,t\Pi{bn,t)\dt- It consists of all functions '^n,ip defined by 

(5.8) ^nAh) = j^^ ^n,tAh)mKt)\dt = 1^ (^^^^W,m(t)|Pf(6n,t)r/VAm;n,t) \mn,t)\dt 

such that the functions (fc^m '■ dn ^ ^ ^^^^ measurable with Ylem\fim{t)\'^ < oo for almost all 
i G T and E^,m I^AmCOl' ^ \Pi{bn,t)\dt). The norms are 

Il*n,,,|li2(^„) = / \\^n,t,Xr.t\PiiKt)\dt= I (V We,r.{t)\^)\mn,t)\dt 

(5 9) ^ ' JT Jt ^ — ^ 

The left /right representation of x Nn on £n,t is the exterior tensor product tt^^j Kl tt* j; it is 
irreducible and the left/right regular representation of Nn x Nn on L'^{Nn) is the multiplicity free 
unitary representation /j*(7r„,t Kl tt* ()|Pf (6n,t)|(ii. 

Let m ^ n. Then |Pf(6„,t)|^/2/^,in;n,t |Pf (W)r^^/^,m;m,t defines an equivariant isometric 
injection ^n,(p{t) '-^ ^m,<^(i) of £n,t into £m,t- The norm computation just above gives 

(^•10) ll*m,|Pf(6„.t)/Pf(W)l'/Vlli2(JV^) = Il*n,<^|li2(jv„)- 

Thus we have an {Nn x A?„)-equivariant isometric injection 

(5.11) Cm,n ■■ L^{Nn) ^ L^{Nrn) by Cm,n(^n,^) = ^m,|Pf(6„,t)/Pf (b,n,t)|i/ V 

On the level of coefficients it is given by C.rn,n{^n,t,ip) = ^m,i.lPf(6„,t)/Pf(6m,t)|i/V' other words 
Cm,n sends the function X)^,m '*^^,m(i)|Pf (&n,t)|^''^/£,m;n,t on Nn to the function on Nm given by 

^(|Pf (6„,t)/Pf (6„,t)|^/V^,m(t))(|Pf (6m,t)r/VAm;m,t) = ^ ¥'€,m(t) |Pf (6n,t) | ^V^.m;*- 

l,m 

The result is 

Theorem 5.12 There is a strict direct system {L?'{Nn),C,m,n] of spaces. The direct system 
maps C,m,n '■ L'^{Nn) ^ L,'^{Nm) O'f^ {Nn X Nn) -equivariant unitary injections. Let Yin denote the 
left/right regular representation of Nn x Nn on Lp'{Nn) and let N = limA^jj. Then we have a well 
defined Hilhert space L?'{N) := lim{L^(A^„), and a multiplicity-free unitary representation 
n = limn„ ofNxN on L'^{N).~^ 
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5B Semidirect Product Groups. 



We need some structural results from ^W5l Section 7] for a strict direct system {Kn, ^Pm,n} of 
compact connected Lie groups and a consistent family {jn} of representations of the Kn on a fixed 
finite dimensional vector space j. We'll use that information to extend Theorems 14.171 and 14.281 to 
a larger family of strict direct systems of nilmanifold Gelfand pairs. 

As just indicated, {Kn, ^m,n} is a strict direct system of compact connected Lie groups. Denote 
K = lhn{Kn,(pm,n} and let {jn} be a consistent family of unitary representations of the Kn on 
a fixed finite dimensional real vector space 3. The C/„ = ^n{Kn) form an increasing sequence of 
compact connected subgroups of dimension ^ (dimj)^ in the unitary group U{^), so from some 
index on they are all the same compact connected subgroup U of U{j). Now we truncate the index 
set and assume that the 7„ all have the same image U in the orthogonal group of 3. 

Let Kn denote the identity component of the kernel of 7„. Then ipm n{Kn) C Km, so we have 
fm.nlf^i}, and K^ is the identity component of the kernel of 7. Since Kn is compact 

and connected, and Kn is a closed connected normal subgroup, Kn has another closed connected 
normal subgroup L„ such that Kn is locally isomorphic to the direct product Kn x L„. 

Proposition 5.13 One can choose the groups Ln so that ifni,n{Ln) = Lm for m ^ n » 0. 

Let L = l\m{Ln,ipm,n\L„}- We further truncate the index set so that L„ = ipn^{L) for all 
indices n. 

Corollary 5.14 Let t E 3, and let Kn,t be its stabilizer in Kn- If one of the direct systems 
{Kn, fm n}, {Kn t, n\K„ At {K^, (fm n\ i>-t I is paraboUc, then the other two also are parabolic. 

Corollary 5.15 Let t G 3, and let Kn,t be its stabilizer in Kn- Suppose that the direct system 
{Kn, fm,n} is paratoHc- Then there are natural isometric injections J^n,t,x ^ ^m,t,\ for m ^ n, 
from the highest weight X representation space of Kn.t to that of Kni,t, md corresponding L^- 

isometric injections f ^ {{d^S '^m,t,x)/{'^^S '^n,t,\))^^'^ f on spaces of coefficient functions - 

Corollary 5.16 Let t G 3, and let Kn,t be its stabilizer in Kn- Then L := limL„ is compact, 

K = K^L where K'^ := lim Kn, and K is locally isomorphic to K'^ x L. In particular K acts on 3 
as a compact linear group and 3 has a ^{K) -invariant positive definite inner product. 

Now we can proceed along the lines of |W5t Section 8]. Fix a strict direct system {{Gn,Kn)} 
of Gelfand pairs that satisfies 

(i) Gn = Nn X Kn-i semidirect product, where Nn is a connected, simply connected, 
nilpotent Lie group with square integrable representations and Kn is connected, 
(5.17) (ii) the Kn form a parabolic strict direct system, 

(iii) the inclusions rin '-^ n„+i map centers in — 3n+i and map complements 0^ ^ On+i, 

(iv) for each n the complement D„ is Ad(^C„)-invariant. 
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Using (j5.17f iii)l we identify each 3„ with 3 := lim3„. Write kJi for the identity component of 
the kernel of the action of Kn on 3. Since Ad(K„)|j is a compact connected connected group 
of hnear transformations of 3, its dimension is bounded, so the Ad(ir„)|j stabihze and we may 
assume that each Ad{Kn)\^ = U for some compact connected group U of hnear transformations 
of 3. Proposition 15.131 gives us complementary closed connected normal subgroups L„ C that 
inject isomorphically under Kn ^ Kn+i, so each L„ is equal to L := limL„. Thus we have 

decompositions Kn = Kn ■ L and K = K'^ ■ L where Kn is the kernel of the adjoint action of Kn on 
I, K = lini Kn , and K^ = lirn Kn- For each n, Ad^^ maps L = Ln onto U with finite kernel. 

If t G 3* denote Ot = Ad*(L)(t), and let Gn,ti Kn^t and Ln^t denote the respective stabilizers of t 
in Gn, Kn and L„. Since Ad*(G„) acts on Ot as the compact group L there is an invariant measure 
ut derived from euclidean measure on 3*. Given t G 3* its stabilizers Gt = {g & G \ Ad*(g)t = t}, 
Kt = KnGt and Lt = LCiGt- Their pullbacks in Gn are Gn,t, Kn^t and Ln^t- Note that Kt = K^ -Lt 
and Kn,t = kI ■ Ln,t- 

Recall T = {t G 3* | each Pf(5n,t) 7^ 0} and fix t G T. Essentially as in the Heisenberg 
group case, the square integrable representation -Kn^t extends to a unitary representation ttJ^ of 
Gn,t ■= Nn XI Kn^t on the same representation space 7in,t- If tin,t,x G Kn^t has representation space 
^n,t,\ we write for its extension to a representation of Gn,t on J-n,t,x that annihilates A^^. 

Then we have the irreducible unitary representations 

(5.18) TT^^j^^ := TTn^t ® /tJ^ G G„,t with representation space 'H^^^^ ■= Kn^t ® -^n.i.A- 
That gives us the unitarily induced representations 

vr„,t,A =Indg;;^(7r^_,,)GG; 

(5.19) r 

with representation space nn,t,x ■= / {'Hn,Ad*{k)t ® •^«,Ad*(fc)<.Ad*(fc)A) dvt{k{t)). 

JOt 

According to the Mackey little group theory, (i) 'Kn,t,x is irreducible, (ii) 'Kn,t,x is equivalent to 
7i"n,t',A' if and only if t' G Of, say t' = Ad*(£)t where £ £ L, and Ad*(£) carries A to A', and (iii) 
Plancherel-almost-all irreducible unitary representations of Gn are of the form TTn,t,\ where t £ T 
and Kn^t,x S Kn,t- So the left/right regular representation of Gn 



n«=E / (>r„,,,AH<,,j)<i(Ad>(A„)(*)) 
(5.20) .^-'-.A'- 

with representation space L'^{Gn) = \] / {Tint x'^T'int x)d{■^(^*{Kn){t))■ 

^ J Ad*(Kn)\i* 



Since Tr^^^^y is square integrable and TTn,t,x is irreducible, one knows ( |W5l Theorem A.l]) that 
Trn,t,x has a well defined formal degree. Making use of Corollaries 15.141 15.151 and 15. 16^ we have 
Gri~equivariant isometric injections 

Cm,n ■ L^{Gn) ^ L^{Gm) based on the Cm,n,f,A : T~(-n,t,X(^'Hn^t,X ^m,t,A§Wm,t,A 

where Cm,n,t,A((deg7r„_t^A)^^^/) = {degiTm,t,x)^^'^ f. 
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With G := lim Gn we now have the 

(5.22) left/right regular representation 11 := limn„ of G x G on L'^{G) := lim{L^(G„), Cm,n}- 

Now we turn to regular functions. As one might expect, £^t\ ™eans t x^^'^^l t \)* viewed 
as matrix coefficients of vr^^ ^. As in the (j4.18p we define 

AiGn t) '■= {finite linear combinations of the e~"*"/^ m i i-n t \ in £'5+ \l) 

A{Gn,t) A{Gm,t) by e "*"/^,m,j,i;n,i,A ^ & ^™ fe,in,i,j;m,t,\, and A{Gt) = lim^(G„,t). 

The norm \ \t\\ on 3* is from a [/-invariant inner product. 

The representation space Tln.t,\ of TTn,t,x = ^^'^cZt^^ntx) space of sections of the 

Hilbert space bundle H^^ ^ ^ Ot = Kn/Kn,t- We express it as the space {L"^ {Kn)®Tn,t,\®'Hn,t)^"''^ 
of i^r„^t-invariants in L'^{Kn) ^^n,t,\®'Hn,t- Let 'H^°l^ denote the space of finite linear combinations 
of the /im in the space 7{n,t- Then the underlying algebraic space is the space A{7rn,t,x)^"'^ '■= 
{A{Kn) ^n,t,A ® T-ilf)^"'' of i^„,t-invariants in A{Kn) J'n,t,x ® . With that and (fCTIl in 
mind we define 

A{Gn) ■■= (finite linear comb of the e-ll*llp(t)/„,t,A where fn,t,x e {A{Kn) ® .F„.t,A ® Hr*^)^-*} , 
(5.24) 1^ 'J 

and ^(G) = lim^(G„). 

where the p{t) are real polynomials on 3*. The maps corresponding to those of (j2.2ip are the 

r]n,t,\ ■■ A{7:n,t,\) £n,t,\ by ??n,t,A(e""*"p(t)/^,m,j,i;t,A) = e""*" dcg 7rn,t,X V{t) f Lm,i,j;t,\ 

Proposition 12.221 now tells us that 

Proposition 5.25 The maps r]m,t,\ satisfy r]m,t,x ° Cm,n,t,x = Cm,n,t,x o r?„,t,A on A{TTn,t,x) and send 
the direct system {A{Gn)} into the direct system {L^(G„), Cm.n}- That system map defines an 
(G X G)-equivariant injection rj : A{G) L'^iG) with dense image. In particular rj defines a pre 
Hilbert space structure on .4(G) with completion isometric to L^{G). 



5C Commutative Quotients. 

We modify the results of Section I5BI to strict direct systems of commutative spaces that satisfy 
(j5.17p . Here the results are recalled from \W5\ Section 9]. 

Theorem 5.26 Let t £ T. Then {Gn.t, Kn,t) is a Gelfand pair. In particular Kn,t is multiplicity 
free on C[ti„]. 

We have already used Hilbert bundle model for the induced representation 'Kn,t,x G Gn,t, given 
by ■Kn,t,x = ^'^'^GZMn,t,x)- ^^^^ representation space 'Hn,t,x of iTn,t,x consists of aU L'^{Kn/Kn,t) 
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sections of the homogeneous bundle p : W^tx ~^ Gn/Gn,t = Kn/Kn,t whose typical fiber is the 
representation space "H^^y^ of T^^tx- Given k G Kn we write k ■ Ti.^^^ for the fiber p~^{kKn,t)- 
Let u E "^^tA * '''"n t a(-^".*)"-^-^^^ ^^^^ vector. Then u belongs to the fiber 1 • 'H'^^^, and 
k ■ u ^ k ■ Ti^^ depends only on the coset kKn,t- Define a section 

(5.27) au:KjKn,t^M^^^ by auikKn,t) = k ■ u. 

Then cr„ is a 7r„ (j^(/C„)-invariant unit vector in the Hilbert space 'Hn,t,\- (We will also write <pu 
for the corresponding function G„, — > such that ^Puiggt) = "^nt xi9t)^^i^u{g)) for G G„, and 

S Gn,t-) Conversely if a is a 7rn^f^A(-f^n)~invariant unit vector in Hn,t,x, then a{lKn,t) = cm where 
|c| = 1 by ii'n.i-invariance, and then a = cGu by X-invariance. In summary, 

Lemma 5.28 Let t £ T and let u be the unique (up to scalar multiple) vr^^ ^{Kn,t)^fixed unit vector 
in 'H'^^y^ Then the section Gu, given by ()5.27p . is the unique (up to scalar multiple) 'Kn,t,x{K) -fixed 
unit vector in 'Hn,t,x- 

By Theorem [526] we can apply dOT]) and ([5:22]) to the function spaces £l^ y^ = 'H''^^t,x^{^n,t,x)* 
on the groups Gn,t = Nn x Kn,t- Making use of Lemma 15.281 we have 

Proposition 5.29 // orthogonal projection S^^x ~^ ^ntx "^e^ds a nonzero right Km,t^i™ariant 
function to a nonzero right Kn^t^invariant function, then orthogonal projection £rn,t,x ~^ £n,t,x sends 
a nonzero right Km^invariant function to a nonzero right Kn-invariant function. 

E. Vinberg classified the maximal irreducible nilpotent Gelfand pairs. See |V1) . |V2j . or see |W3[ 
Table 13.4.1]. A Gelfand pair {Gn,Kn) is called maximal if it is not obtained from another Gelfand 
pair (G^,K^) by the construction {Gn,Kn) = (G'^^/C, K!^/{K'^ n C)) for any nontrivial closed 
connected central subgroup C of G'^. And {Gn,Kn) is called irreducible if Ad{Kn) is irreducible 
on tJn = n„/3. See |W3| for the notation. 



(5.30) 



Maximal Irreducible Nilpotent Gelfand Pairs {N„ >i Kr,,Kn) (Wli. tV2i) 




Group Kn 


On 


3 


(7(1) is 

nnr>dnrl if 


max 
requires 


1 


SO{n) 


K" 


SkewR"^" =so(n) 






2 


Spin{7) 


K« = 


= Im 






3 


G2 


= Im 


= Im0 






4 


U{l)-SO{n) 


C" 


ImC 




n 7^ 4 


5 


{U{l)-)SU{n) 


C" 


A^C" ® Im C 


n odd 




6 


S'[/(n),n odd 


C" 


A^C" 






7 


SU{n),n odd 


C" 


ImC 






8 


U{n) 


C" 


ImC"^" = u(n) 






9 


{U{l)-)Sp{n) 


H" 


ReHJf^"©ImH 






10 


U{n) 




M 






11 


{U{l)-)SU{n),n'^ 3 


A^C" 


M 


n even 




12 


[/(I) • Spin{7) 


C« 


M'^eM 






13 


U{1) ■ Spin{9) 




M 






14 


{U{l)-)Spin{lO) 




M 






15 


U{1) ■ G2 




M 







.... table continued on next page 
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.... table continued from previous page 



(5.30) 



16 


U{1) ■ Ee 




M 






17 


Sp{l) X Sp{n) 




Im]H = sp(l) 




n ^ 2 


18 


Sp{2) X Sp{n) 




ImM^^^ =sp(2) 






19 


{U{l)-)SU{m) X SU{n) 
m,n ^ 3 




R 


m = n 




20 


{U{1)-)SU{2) X SU{n) 




ImC^x^ = u(2) 


n = 2 




21 


{U{iy)Spi2) X SU{n) 




M 


n ^ 4 


n ^ 3 


22 


U{2) X Sp{n) 




ImC^><^ = u(2) 






23 


U{?,) X Sp{n) 




R 




n ^ 2 



Splitting some cases to retain parabolicity of {K^}, the strict direct systems in Table [5^30] with 
dim3n bounded, are 
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Direct Systems of Maximal Irreducible Nilpotent Gelfand Pairs {Nn x Kn,Kn) 




Group Kn 


On 




1/(1) is 
nendnd if 


max 
requires 


4a 


U{1) • SO{2n) 




ImC 




n / 2 


4b 


U{1) ■ S0{2n + 1) 




ImC 






7 


SU {n),n odd 




ImC 






10 


U{n) 




R 






11 


{U{iy)SU{n),n ^ 3 




R 


n even 




17 


Sp{l) X Sp{n) 


H" 


ImE[ = sp(l) 




n ^ 2 


18 


Sp{2) X Sp{n) 




ImH^^^ =sp(2) 






19 


{U{l)-)SU{m) X SU{n) 
m,n ^ 3 




R 


m = n 




20a 


SU{2) X SU{n),n^ 3 




ImC^^^ = u(2) 






20b 


U{2) X SU{n) 




ImC^^^ = u(2) 






21 


{U{l)-)Sp{2) X SU{n) 




R 


n ^ 4 


n ^ 3 


22 


U{2) X Sp{n) 




ImC^^^ = u(2) 






23 


U{3) X Sp{n) 




R 




n ^ 2 



In each case of Table [STST] |W31 Theorem 14.4.3] says that Nn has square integrable represen- 
tations. In the cases dim 3 > 1 of Table [HTHT] we have Kn = Kn ■ L where the big factor Kn acts 
trivially on 3 and the small factor L acts on 3 by its adjoint representation. Summarizing these 
observations, 



Proposition 5.32 Each direct system {[Gn,Kn)} of Table 15.311 has the properties (i) {Kn} is 
parabolic, (ii) the {Kn,t} CLre parabolic for each t £ T, and (iii) Nn has square integrable represen- 
tations. 



From Theorem 15.261 and the argument of Lemma 14.201 
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Corollary 5.33 Let {{Gn, Kn)} be a direct system of Table 15.311 and let t ^T. Then vr^^^ has a 
nonzero Kn,t-fixed vector if and only if k*^ix occurs as a subrepresentation of TT^t\K„,t' ^'^d in that 
case the space of Kn,t~fi-xed vectors has dimension 1. 

Now everything goes essentially as in the Heisenberg nilmanifold cases of Section [H We have 
isometric G„-equivariant injections 

(5.34) Cm,n,i,A : £n;^x ^ ^m7,X Cm,n,t,\iideg7rr,,t,\)^/^ f) = C™,„,t,A(deg 7r^,i,A)^/V 

where as in (|4.24p . Cm,n,t,\ is the ratio (|4.23p of lengths of Kn, Km fixed unit vectors. Integrating 
on t and summing A gives isometric G^-equivariant injections Cm,n ■ L'^{Gn/ K^) — > L?'{Gm/ Km)-, 
as follows. 

Theorem 5.35 For each of the direct systems of Table 15.311 denote G = lim{G„,}, N = limA^„ 
and K = lim{i^„}. Note G = N xi K . Then {L'^{Gn/Kn),C,m,n} is a strict direct system of 
Hilbert spaces in which the maps Cm.n ■ L'^{Gn) ~^ L'^{Gm) Gn^equivariant injections. That 
gives us the left regular representation of G on the Hilbert space L'^{G) := lim{L^(G„), Cm,n.}- 
Further, that left/right regular representation is the multiplicity-free "^^^ lAd*{L)\f (■^Ad*(fc)t,Ad*(A;)A^ 
^Ad-(fe)t,Ad*(fc)A) dk where TTt,x := lim7r„,t,A. 

Similarly using Wm,\ = Cm,n,t,\Wn,\ + x with x _L £n,t,\ with < Cn,t,\ S 1, 

(5.36) A{Gn/Kn) := A{Gnf" = A{Gn) n £^l^ 
leading to direct systems and their limits by assembling the maps 

(5.37) fm,n,t,A : A{TTn,t,x)^" ^ A{lTm,t,\)^"" by fu,v„,n ^ fu,v^,m where Pm,n,t,\iVm) = Vn 

Then we have direct systems and their limits 

A{G) = lim{^(G„), Cm,n} where Cm,n,t,\ ■ £n,t,x ^ £m,t,\ (i5.2ip, and 
A{G/K) = lim{^(G„)/i^„), i/„,4 where VmM ■ S^Jlx ^ £^J,x dS^ZD- 

As before, each A{Gn/Kn) is dense in Lp'{Gn/ Kn), and we pass this comparison to the limit 
with the maps 



(5.39) rin,t,x ■■ A{i:n,t,x) ^" ^ '^^r.,t,x ® {wn,t,x*C) by f ^ Cn,i,t,x v^iegTr^/- 

We conclude 

Proposition 5.40 The rjn,t,x satisfy {rjm,t,x ° ^m,n,t,x){f) = {Cm,n,x ° rjn,t,x){f) for f G AiTTn^t^)^"" 
Thus they inject the direct system {A{Gn/Kn),i^m,n} of regular functions into the direct system 
{L'^{Gn/Kn),Cm,n}- That map of direct systems defines a G-equivariant injection r/ : A{G/K) 
LP'{G/K) with dense image. In particular rj defines a pre Hilbert space structure on A{G/K) with 
completion isometric to L?'{G/K). 
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5D Reducible Quotients. 

There are more many strict direct sequences of nilmanifold Gelfand pairs {Gn,Kn), e.g. those 
for which the action of Kn on nn/jn is reducible. These [Gn/Kn) are constructed from certain 
basic ones that satisfy several technical conditions (indecomposable, principal, maximal and 5*^(1)- 
saturated). See |Y1] . |Y2| . |Y3j . |W3| and |W5] . The basic such direct systems, with Kn reducible 
on nn/ln^ dim^n bounded and {Kn} parabolic, are tabulated in [W5t Table 9.15] as follows. Here 
the numbering comes from jW5l Table 9.14], Ni = x Zi with Zi central and maximal for that, 
Gi = Nf Xi Kf and G'f = xi K^ . The generalized Heisenberg algebra t)n.¥ is Im F + F" with 
[{z, w), {z' ,w')] = (z + z' + Im {w, w'),'W + w') where F = C, EI (quaternions) or (octonions). 



(5.41) 

Strict Direct Systems {{Gi,Ki)} and {{G'g,K'g)} of Gelfand Pairs with dimj^ Bounded 





Group Ki 


ii'^-niodule Vg 




3'/ 


Algebra 


1 


U{n) 


C" 


M 


Su(7l) 


f}n;C 


3 


U{1) X U{n) 


C" ® A^C" 


M® K 





l}n;C ® f)ri(n-l)/2;C 


6 


S{U{4:) X U{m)) 




M 


E'' 


[}4m;C 


7 


U{m) X Uin) 




M ® E 





t)mn;C ® f)m;C 


8 


U{1) X Sp{n) X U{1) 


C^" ® C^" 


M ® E 





f}2n;C ® f)2n;C 


9 


Sp{l) X Sp{n) X U{1) 


H" ® H" 


IniH® K 





f}n;H ® f)2n;C 


10 


Sp{l) X Spin) X Sp{l) 


H" ® H" 


IniH® ImH 





t)n;H ® f)n;H 


11a 


Sp{n) X Sp{l) X Spim) 


H" 


ImH 


Jplj-n X 771 


t)n;H 


lib 


Sp{n) X U{1) X Sp{m) 


H" 


ImH 


l^n X rn 


t)n;H 


11c 


Sp{n) X {1} X Sp{m) 


H" 


ImH 


jjH^n X m 


f|n;H 


18a 


SU{n) X SU{2) 




E 


su(2) 


[)2« 


C 


18b 


U{n) X SU{2) 




E 


su(2) 


[)2« 


C 


18c 


C/(l)S'p(f ) X 5[/(2) 




E 


su(2) 


[}2« 


c 


19a 


SU{n) X U{2) 


C"^^ ® 


E® E 





t)2n 


C® f)2 


C 


19b 


U{n) X U{2) 


C"^^ ® 


E® E 





[}2n 


C® f)2 


C 


19c 


U{l)Sp{^^) X U{2) 


C"^^ ® 


E® E 





[)2n 


C® f)2 


c 


20aa 


SU{n) X SU{2) X SU{m) 


(j~.nx2 ^£.2xm 


E® E 





t)2n 


C ® f)2m 


c 


20ab 


SU{n) X SU{2) X U{m) 


^nx2 ^ (^2xm 


E® E 





t)2n 


C ® f)2m 


c 


20ac 


SU{n) X SU{2) X U{l)Sp{f) 


(^«X2 (^£.2xm 


E® E 





[)2n 


C ® f)2m 


c 


20ba 


U{n) X SU{2) X SU{m) 


^nx2 ^ ^2xm 


E® E 





[)2n 


C ® f)2m 


c 


20bb 


U{n) X SU{2) X U{m) 


^nx2^f^2xm 


E® E 





t)2n 


C ® f)2m 


c 


20bc 


U{n) X SU{2) X U{l)Sp{f) 


^nx2 ^ (^2xm 


E® E 





[)2n 


C ® f)2m 


c 


20ca 


U{l)Sp{^) X SU{2) X SU{m) 


^nx2 ^^2xm 


E® E 





[)2n 


C ® f)2m 


c 


20cb 


U{l)Sp{^) X SU{2) X U{m) 


^nx2 (^^2xm 


E® E 





[)2n 


C ® f32m 


c 


20cc 


U{l)Sp{^) X SU{2)x 

U{l)Sp{^)} 


£.«X2 (g£.2xm 


E® E 





t)2« 


C ® f)2m 


c 


21a 


SU{n) X SU{2) X U{4) 


^nx-Z^^ZXi 


E® E 


E« 


[)2«;C ® t)8 


c 


21b 


U{n) X SU{2) X U{4) 


(J-«X2 (^£.2X4 


E® E 


E« 


[)2n;C ® t)8 


c 


21c 


C/(l)S'p(f ) X SUi2) X C/(4) 


£.«X2 ^£.2X4 


E® E 


E« 


l)2n;C ® f)8 


c 



One obtains the structure of L'^{G/ K) and A{G/ K) for the cases of Table [574T] by a straightforward 
modification of the considerations involved for the direct systems of Table We leave the details 
to the reader. 
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